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WTien we proposed the topic for this paper last August, we had already 
spent a prestigious five weeks in the field of instructional materials devel- 
opment and evaluation. The survey we were proposing seemed to be a use- 
ful and necessary step in the assessment of the state of the art in instruc- 
tional materials development and an interesting prologue to the two succeeding 
papers in today's symposium. We soon found to our chagrin, however, that 
such a survey could not be reasonably undertaken given the current state of 
the art in product development. We, therefore, turned our attention to the 
task of assessing just where we are currently with respect to the use of 
models in instructional product development and evaluation and to the task 
of identifying the antecedent conceptual problems which must be resolved 
before much productive research in the area can be undertaken. We found 
ourselves in the same position as Norman Hamm in his discussion of the 
efficacy and administration of research on the mental health of children 
(Hamm, 1974). He notes that his method of inquiry is 'limited, if not 
somewhat paradoxical. What is offered is a rational analysis of our empir- 
ical enterprise as it applies to practical payoff..." p. 9. We join him in 
calling for more attention to a science of science including research per- 
formed on research policy. 

As a first step, then, we need to examine just what is meant by the 
use of evaluation and development models in product development. 



PuriDOses of Mode ls 

Individuals writing on the nature and use of models in science (c.f. 
Kaplan, 1964, Black, 1973) have noted several uses made of the term. 
Black, for example, discusses how the term 'taodel" has been used in 
science to denote (1) scale models, (2) analogue models in which there 
is an isomorphic relationship between the model and the original entity, 
(3) a model which is a type of design of something worthy of imitation, 
and (4) theoretical models in which one uses an area better understood 
in order to solve a particular problem. We will argue that the term 
''models** as typically used in educational development and evaluation 
appears to refer to the third definition above, i.e., these development 
and evaluation models are simply designs of exemplar processes or pro- 
cedures. They do not appear, for instance, to be theoretical models 
since they generally lack explanatory and/or predictive features. They 
simply appear to be examplars of desirable or commendable operating 
procedures and are probably best viewed as alternative, descriptive 
conceptualizations of development and evaluation processes. Black men- 
tions such a use when he discusses models as a "convenience of e:q)0- 
sition. . . /that/ may also help us to notice what otherwise may be over- 
looked, to shift the relative emphasis attached to details — in short, 
to see new connections. " p. 496. This use of the term "models" seems 
particularly appropriate to the models used to guide evaluation design 
where there are several competing conceptualizations of approrpiate 
evaluation procedures, each based on differing assumptions and values 
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concerning the evaluation process itself. Development models, however, 
frequently appear to be as much managerial strategies as helpful concep- 
tual paradigms of any particular development process. The characteriza- 
tion of models as basic conceptual devices for heuristic use as appropriate 
in particular settings suggest at this point, at least, a conceptual difficulty 
in generalizing or being prescriptive about the use of particular models 
across varying circumstances and situations. 

Reviews of Models 

Although space does not permit a full review of all development and 
evaluation models currently being proffered, a few words of summary are 
included. 

Probably the most current and comprehensive discussion of models 
and procedures for instructional development can be found in Baker (1973). 
Both Baker and Schutz (1970) have referred to two major development 
emphases: those having to do with a change support process, and those 
having to do with a product development process. They suggest that there 
may be differing development models and strategies appropriate to the dif- 
ferent emphasis of these two approaches. Baker provides some reason to 
doubt whether there could be any generalizable development models for 
change support processes in pointing out that development procedures are 
highly idiosyncratic to particular contexts* Merrill and Boutwell (1973), 
in reviewing books and manuals 'designed to give guidance to developers of 
instructional systems, have noted that though they vary in terminology, the 
procedures recommended all basically include the following components: 



behavioral objectives, pre-test, instructional activities, post-test, and 
re\dsion based on empirical results. 

There appears to be less uniformity in the field of evaluation models 
with its proliferation of models such as the CIPP Model (Stufflebeam, 19G8), 
the Countenance Model (Stake, 1967), the Discrepancy Model (Provls, 
1971), the Pathways Model (So riven, undated), etc, Worthen and Sanders 
(1973) provide the most current comparison of the major evaluation 
models, though the collection of readings by Taylor and Cowley (1972) also 
provides a good overview. 

In addition to the increased attention being paid by the research com- 
munity to the development and use of models, there is also apparently an 
increasing tendency to use models as a means of specifying and operation- 
alizing federal development and evaluation policy. Recent drafts of policy 
statements from the National Institute of Education (NIE) (c.f. NIE 1974a, 
NIE 1974b) and documents submitted to NIE for possible policy adoption, 
(c.f. Institute for Educational Development, 1973) suggest that NIE is 
taking a more prescriptive stance toward the use of evaluation and devel- 
opment models in its monitoring of instructional development work. 

This increased prescriptiveness seems to emanate, at least in part, 
from NIE's response to a report to Congress made by the Comptroller 
General concerning a review of educational laboratory and research devel- 
opment center programs (Comptroller General of the United States, 1973). 
In its response to this report, NIE agrees, among other things, to (1) 
require contractors to state objectives in terms of the specific educational 



changes that are expected, (2) to produce empirical evidence of the accom- 

plislmients of students who use the materials, and (3) to define more 

clearly the various stages of product development and testing (Comptroller 

General of the United States, 1973). 

By NIE's own admission this increased prescriptiveness is not based 

on empirical evidence. 

"Generally we have lacked both the data base and the understanding 
of system dynamics needed for effective, rational policy making. In their 
absence, policies have been determined on the basis of the resolution of 
cross-pressures, analogies drawn from other fields, and the imagination 
and foresight of a few individuals. " (NIE, 1973, p. 65) 

Just how true this statement must be Is evidenced by a brief review of the 

past research in the area of evaluation and development models. 

Research on Models 

Whether one looks at the overall rationale and empirical evidence 
supporting the use of development models or the justification of specific 
components of development procedures, the picture is equally bleak, 
Merrill and Boutwell (1973), after noting that almost all development pro- 
cedures reviewed seemed to be based upon (a) the specification of objectives 
in terms of observable student behaviors, (b) the use of criterion-referenced 
rather than norm-referenced measurement procedures, and (c) the verifica- 
tion of instructional products by empirical test results, note that "the paucity 
of research on these premises suggest that they are considered axiomatic. 
Numerous propositions have been suggested, but only a handful of rather 
poorly executed studies have tested them*" p. 95. Baker (1973), also 
notes that much of the literature in instructional development today prescribes 



on the basis of faith alone, "legions of procedures. " Noting a lack of em- 
pirical research in this area, she summarizes by saying, 

''...the field of educational development requires inquiry into the 
kinds of development procedures which are effective in particular contexts 
for various classes of organizations. From the current state of the art, 
one would suppose that minimally such procedures would require specifi- 
cation, field testing and revision as a foundation for development work. 
The possible ways in which each of these points might be translated into 
practice must be explored. " p. 277. 

Though the specification of behavioral objectives may be thought to be 
basic to most development models, the results of research on the effective- 
ness of using behavioral objectives as a means of improving teaching, 
guiding evaluation, or in the facilitation of learning, seems to be inconsis- 
tant and equivocal at best (Duchastal and Merrill, 1973; McKeachie, 1974; 
Stake, 1973). There are a number of conceptual and technical challenges 
which the criterion referenced test movement must meet before any devel- 
opment model could safely depend upon their wide spread development and 
use (Ebel, 1973; Klein and Kosecoff, 1973). Further, Baker (1973) indicates 
that there is not even indicative evidence that certain types of media con- 
tribute to a better learning environment than other types of media or no 
media at all. She suggests that developers themselves have to make 
evaluation studies of probably contending media to determine their relative 
effectiveness for the particular objectives of interest. And finally, it is 
becoming increasingly clear that the adoption of even the most basic 'laws 
of learning" and hierarchial relations which has been a very significant 
feature of development models is largely without empirical justification as 
they are now conceptualized (McKeachie, 1974a, McKeachie, l974b. White, 
1973). 



A look at the move global components of development models reveals 
the same lack of research foundation. Baker (1973) notes that while there 
are many rjiiides for writing performance specifications, the research in 
support of such practice is not to be found. She indicates that the influence 
of specification on the purveyors of instruction has not been demonstrated 
in controlled experimental studies. Although there are many guidelines on 
how to write performance specifications, there is no research today to 
indicate the utility of such procedures to other developers. She notes that 
the question of the generalizability of specific revision procedures to other 
tasks and other subject matters have only been touched upon briefly. 

Popham (1969) notes that this same situation prevails with research 

concerning the development of curriculum materials. 

. . studies for the revision process to improve the quality of curri- 
culum materials have not been frequently conducted. There seems to be 
an overriding faith in the idea that materials revised to be consistant with 
empirical tryout data will become better. This has not been clearly demon- 
strated. Certainly the maimer in which revisions can be made most 
efficiently has not been carefully treated. " p. 335. 

The state of the art appears to be no better for evaluation models. 
Worthen (1972) has pointed to the sparcity of research on the phenomena 
of evaluation itself. He notes that present evaluation models "...repre- 
sent (individually and collectively) an inadequate knowledge base for the 
guidance of educational evaluation." p. 3. He notes there is little or no 
information about the relative efficacy of alternative evaluation techniques 
and calls for research on the evaluation models themselves as well as on 
the critical components of most models, such as the obtrusiveness of data 
collection techniques on the educational phenomena being evaluated, and the 



differential effectiveness of feedback provided from evaluation under differ- 
ing conditions and scheduling. He also notes the need for studies to 
identify the most effective means of identifying goals and conducting needs 
assessments. 

Stake and Denny (19G9) have written about the lack of available tech- 
niques and knowledge base for the conduct of evaluation. They discuss, 
for example, the lack of techniques for presenting goals and priorities and 
the relative lack of good instrumentation for assessing instructional mater- 
ials in classroom instruction. For example, acknowledging that there are 
many traditional observational schedules and techniques, they point to how 
little information is available on how validly these techniques can be used 
and how widely. In addition, they note the lack of adequate instructional 
assessment schedules and call for research and development which pays 
attention to the ..widely reported phenomenon of experimenter bias and 
the obtrusiveness of the classroom observer. " p. 38. They also indicate 
a need for simple ways of meaningfully describing what the instructional 
program consists of and what the students do in and after training. 
Finally, they discuss evaluation's current lack of ability to show how stu- 
dents perform and to measure student's performance in evaluation- related 
contexts* 

Thus, in evaluation as well as development, there is not only a 
lack of research evidence on the utility of various models or conceptual 
approaches, but a lack of evidence on the effectiveness of component 
procedures and the basic assumptions within the models themselves. It 
appears that, at this time, these models can not be considered anything 



more than convenient conceptual mechanisms or, at most, value-based 
(not empirically based) claims about what constiti.tes "good" development 
and evaluation procedures. 

There may be a need at the federal policy level for increased pre- 
scriptiveness of the use of evaluation and development models, but as 
long as there is such a lack of research evidence to support this pre- 
scriptiveness, it ought to be made clear that such recommendations are 
made from policymakers' past experience or value basis or from untested 
hypotheses concerning the appropriateness of models selected from fields 
other than education and not from empirical based research. 

Implications for Research 

The case has been made thus far that a great deal of research needs 
to be conducted upon evaluation and development procedures before empir- 
ically based policy can be developed. There are some basic conceptual 
problems, however, which must be solved before adequate research can 
be conducted upon the development and evaluation models themselves. 

One basic problem is the conceptual delineation of just what consti- 
tutes a development or evaluation model and how one can identify and 
adequately describe such models. Merrill and Boutwell (1973) have dis- 
cussed the lack of the systematic identification of the variables being 
manipulated by instructional developers and have proposed a taxonomy 
which would enable such identification. They note that it is often difficult 
for instructional material developers to describe their products unambig- 
uously and thereby profit by the work of others in the field. In addition. 



they cite the ditficulties instructional researchers have in describing their 

experimental treatments adequately. Merrill and Boutvvell have proposed 

a task variable taxomony as a first step in resolving some of these 

difficulties. Schutz (1970) notes that 

"The scope of the phenomenon, 'educational development, ' has yet 
to be realized. WTiile there is a vague realization that it should be 
possible to translate available relevant knowledge into a form that per- 
mits improved educational practice, it is erroneous to assume that the 
means for doing this are perfectly clear." p. 39. 

Schutz further argues that the realm of educational development is still 

too young and rapidly growing to make it even feasible to attempt to 

describe the state of the art. 

'While the development domain is highly sophisticated and well dif- 
ferentiated in many areas, in education it has until very recently either 
been regarded as unnecessary or confounded with research. " p. 40. 

That this problem of lack of conceptual clarity is not confined to develop- 
ment is evidenced in the discussion by Worthen and Sanders (1973) of the 
continuing disagreement within the field about the most appropriate and 
useful definition of evaluation, (c.f. Scriven, 1972). 

A second major problem to be overcome before addressing such con- 
cerns as (1) the utility of various development and evaluation models, (2) 
the compatibility of given evaluation and development models in a particu- 
lar context, and (3) the interaction of particular evaluation and development 
models with the nature of the product, is the lack of conceptual clarity 
about the actual purpose for using development and evaluation models in 
materials development. An answer to the question: "What is an effective 
development or evaluation model?" requires that one first consider "effec- 
tive for what purpose?" Models have been used frequently in the past and 



apparently without a need on the part of the users for a strong empirical 
base. Development and evaluation models may currently be being used 
because of a belief (thcro is obviously no empirical evidence) that they 
will result in the efficient production of high quality products and result 
in efficient and relevant assessment of the worth of these products. This 
view of the use of models, however, is based upon an educational tech- 
nology perspective and it is possible that models are used not for their 
utility in terms of educational technology, but because they serve as 
efficient political and managerial means of directing and controlling devel- 
opment work. 

Micheal Apple (1972) has suggested that systems management pro- 
cedures (in many ways similar to the current use of develop icnt and 
evaluation models) are actually means of increasing certainty and control: 

"It should be made clear, then, that systems approaches are not 
essentially neutral, nor are they performing a "scientific" function. By 
tending to cause its users and the other publics involved to ignore cer- 
tain possible fundamental problems with schools as institutions, systems 
management also acts to generate and channel political sentiments suppor- 
tive of the existing modes of access to knowledge and power." p. 15. 

Obviously models may be chosen by researchers for their heuristic 
and conceptual feati^res but used by administrators because of their moni- 
toring and management capabilities. If one assumes that these models 
are being prescribed at the federal policy level, however, more for their 
managerial capabilities than their educational technology base, then the 
previous literature review is incomplete. The criteria for good develop- 
ment and evaluation models then lie not only in their utility in providing 
the most technically sound products, but in their utility for monitoring and 



guiding the development process itself as well. This leads one to view 
the use of models as another type of management strategy and suggests 
research te\aews and further research in the areas of industrial and ed- 
ucational management and not in the area of instructional technology. 

The appropriate criteria and even most germane fields of research 
for development and evaluation models are therefore inextricably tied to 
the purpose or use that such models have in the instructional development 
process. Presumably models can fulfill multiple purposes or roles in the 
instructional development process, indeed So riven (1967) sometime ago 
discussed the multiple roles evaluation can ganerally play. If, however, 
development and evaluation models are simply alternative heuristic con- 
ceptualizations of their respective processes, then the assessment of their 
validity or utility must be made in terms of the value perceptions of the 
relevant individuals involved, i.e., one conducts value research. If, on 
the other hand, these models are to be used as validated procedures for 
producing the technologically best instructional products, then considerably 
more research needs to be conducted in the field of instructional technology. 
Finally, if these models are to be used as a managerial control and moni- 
toring mechanism then research in the area of industrial and educational 
management is most relevant. At present there appears to be little clarity 
in the field of instructional materials development regarding the actual 
purposes development and evaluation models are being used for and more 
importantly, the actual purposes for which such models were orginally 
designed. 



Ill summary, then, it appears that there is practically no solid 
empirical evidence to justify the prescriptive use of development and 
evaluation models in instructional products development — at least from 
an instructional technology standpoint. Furthermore, before such a 
research base eould be provided, two significant conceptual problems 
have to addressed. First, how does one define, identify and describe 
development and evaluation models per se, and secondly, how does 
one determine when an instructional technology perspective is indeed 
more relevant than an industrial management or value research per- 
spective? Perhaps when these questions have been adequately answered 
someone will be able to reasonably conduct a survey of extant develop- 
ment and evaluation models in labs and centers. 
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